Systems and methods for facial expression tracking

ABSTRACT

A facial tracking device including an illuminator and a photon detector. The illuminator configured to project a light toward a first portion of a head of a user and the photon detector configured to detect light reflected from a second portion of the head of the user. The facial tracking device further including a processor that is connected to the illuminator and photon detector and configured to cause the illuminator to project the light toward the head of the user, receive information from the photon detector, and determine a facial expression of the user.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to a facial tracking system.More particularly, the present disclosure relates to systems and methodstracking facial expressions.

BACKGROUND

The present disclosure relates generally to facial tracking systems.Facial tracking systems or devices may be utilized in a variety ofelectronic devices. For example, a facial tracking system may beutilized to determine a facial expression on a face of a user of avirtual reality (VR), augmented reality (AR), or mixed reality (MR)system.

SUMMARY

The subject matter of this disclosure is directed to systems and methodsfor facial expression tracking. In some embodiments, the system includesan illuminator, a single or group of photon detectors, one or moreanalog circuits, and/or a software algorithm to reconstruct facialexpressions based on the information from the photon detectors. In someembodiments, the photon detectors and/or the illuminators do not requirelenses. The system provides a low cost, low computational overheadfacial recognition system that does not require cameras or complex imagedata processing in some embodiments.

One implementation of the present disclosure is related to a wearabledisplay (e.g. a head mounted (HMD)), according to some embodiments. Thewearable display may include an element configured to be worn on a headof a user, an illuminator disposed with the element and configured toemit light toward at least a portion of the head, a photon detectordisposed with the element and configured to receive light reflected fromat least a second portion of the head, and one or more processorscoupled to a non-transitory computer-readable storage medium havinginstructions encoded thereon that, when executed by the one or moreprocessors, cause the one or more processors to cause the illuminator toproject the light toward the portion of the head, receive informationfrom the photon detector, and determine a facial expression of the userbased at least on the information from the photon detector. In someembodiments, the element is a glass frame and/or a housing.

In another implementation, a facial tracking system includes anilluminator configured to emit light toward at least a portion of a faceof a user, a photon detector configured to receive light reflected fromat least a second portion of the face, and a processor. The processor isconfigured to receive information regarding the characteristics of lightdetected by the photon detector, and determine a facial expression ofthe face based on the characteristics of light detected by the photondetector.

In another implementation, a method of facial tracking includesprojecting light using an illuminator of a facial tracking system towarda face of a user, receiving, at the processor, information from a photondetector, the information comprising characteristics of the lightreflected from the face, and determining, via the processor, a facialexpression of the face based on the information from the photondetector.

These and other aspects and implementations are discussed in detailbelow. The foregoing information and the following detailed descriptioninclude illustrative examples of various aspects and implementations,and provide an overview or framework for understanding the nature andcharacter of the claimed aspects and implementations. The drawingsprovide illustration and a further understanding of the various aspectsand implementations, and are incorporated in and constitute a part ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component can be labeled inevery drawing. In the drawings:

FIG. 1 is a block diagram of a facial tracking system in accordance withan illustrative embodiment.

FIG. 2 is a view of a user wearing a head mountable device integratedwith a facial tracking system is shown in accordance with anillustrative embodiment.

FIG. 3 depicts an example of a facial tracking device in accordance withan illustrative embodiment.

FIG. 4 depicts an example of a facial tracking device in accordance withan illustrative embodiment.

FIG. 5 depicts a diagram of a facial tracking device detecting an objectin accordance with an illustrative embodiment.

FIG. 6 depicts a diagram of a facial tracking system in accordance withan illustrative embodiment.

FIG. 7 depicts a block diagram of circuitry of a facial tracking systemin accordance with an illustrative embodiment.

FIG. 8 depicts an example of a facial tracking device in accordance withan illustrative embodiment.

FIG. 9 depicts an example of a facial tracking device in accordance withan illustrative embodiment.

FIG. 10 depicts a diagram of a facial tracking system in accordance withan illustrative embodiment.

FIG. 11 depicts a schematic diagram of a winner take all circuit andexample outputs in accordance with an illustrative embodiment.

FIG. 12 depicts a flow diagram of a method of facial tracking inaccordance with an illustrative embodiment.

DETAILED DESCRIPTION

Referring generally to the FIGURES, systems and methods for a facialtracking systems are shown, according to some embodiments. The facialtracking system includes a facial tracking circuit. The facial trackingcircuit includes an illuminator and a photon detector. In someembodiments, the facial tracking circuit includes two or moreilluminators or photon detectors. In some embodiments, the facialtracking system includes a first facial tracking circuit positioned tomonitor a first portion of a face of a user and facial tracking circuitpositioned to monitor a second portion of the face of the user. In someembodiments, the facial tracking system may include additional facialtracking circuits, each configured to monitor respective portions of theface of the user. In some embodiments, a first illuminator is configuredto project a light toward a first portion of the face (e.g., a rightside of a mouth of the user) of a user and a first detector isconfigured to receive the light reflected from the first portion of theface and a second illuminator is configured to project a second lighttoward a second portion of the face of the user (e.g., a left side of amouth of the user) and a second detector is configured to receive thesecond light reflected from the second portion of the face. The lightreceived at the first detector and the second light received at thesecond detector may then be used to determine a distance that therespective portions of the face are from the detector. The facialtracking system may then determine that the user is making a facialexpression (e.g., a smile, frown, excitement, surprise, etc.) based onthe known portions of the face that the first and second detectors arereceiving the respective light from. In this way, the facial trackingsystem may track the facial expressions of a user with one or moreilluminators and photon detectors, which may reduce the facial trackingsystem's processing requirements and/or footprint. In other words, thefacial tracking system provides for a low cost, low computationaloverhead facial recognition system that does not require cameras orcomplex image data processing.

In some embodiments, the facial tracking system may be implemented invarious form factors. For example, the facial tracking system may beimplemented in the form of worn device such as a head mountable deviceor other device that is positioned to monitor a face of a user. In someembodiments, the worn device may include glasses, goggles, or other headwearable device. Further, the facial tracking system may be implementedor connectable to various electronic devices. For example, the facialtracking system may be implemented with a virtual reality (VR) system,augmented reality (AR) system, or mixed reality (MR) system. In someembodiments, the facial tracking system may be implemented withelectronic devices such as game consoles, server systems, personalcomputers, or gaming systems.

Referring now to FIG. 1, a block diagram of a facial tracking system 100is shown. The facial tracking system 100 may include a facial trackingdevice 101, and processing circuitry 102. The facial tracking device 101includes an illuminator 110 and a photon detector 111. In someembodiments, the facial tracking device 101 may include two or moreilluminators 110 and/or two or more photon detectors 111. In someembodiments, the illuminator 110 may include an infrared illuminator,diode, light emitting diode (LED), lamp, laser, a combination thereof,or other illumination sources. Additionally, the photon detector 111 mayinclude photoconductive detectors, photovoltaic detectors, p-n junctionphotovoltaic detectors, p-i-n photodiodes, avalanche photodiodes (APD),Schottky photodiodes, photo-electromagnetic (PEM) detectors, quantumwell infrared photodetectors (QWIP), a combination thereof, or othertypes of photon detectors. The illuminator 110 is positioned or arrangedto project a light toward a first portion of a face of a user, and thephoton detector 111 is positioned or arranged to receive light reflectedfrom the first portion of the face of the user. In this way, the facialtracking system 100 may monitor movements and/or detect positions ofobjects within the first portion. In some embodiments, the facialtracking system 100 and/or facial tracking device 101 is part of a gameconsole, a VR, AR, or MR console, or a wearable display.

The processing circuitry 102 may include a processor 120, a memory 121,and an input/output interface 122. In some embodiments the processingcircuitry 102 may be integrated with various electronic devices. Forexample, in some embodiments, the processing circuitry 102 may beintegrated with a wearable device such as a head worn display, smartwatch, wearable goggles, or wearable glasses. In some embodiments, theprocessing circuitry 102 may be integrated with a gaming console,personal computer, server system, or other computational device. In someembodiments, the processing circuitry 102 may also include one or moreprocessors, application specific integrated circuit (ASICs), orcircuitry (e.g., such as winner take all circuitry) that are integratedwith the facial tracking device 101 and are designed to cause or assistwith the facial tracking system 100 in performing any of the steps,operations, processes, or methods described herein.

The processing circuitry 102 may include one or more circuits,processors 120, and/or hardware components. The processing circuitry 102may implement any logic, functions or instructions to perform any of theoperations described herein. The processing circuitry 102 can includememory 121 of any type and form that is configured to store executableinstructions that are executable by any of the circuits, processors orhardware components. The executable instructions may be of any typeincluding applications, programs, services, tasks, scripts, librariesprocesses and/or firmware. In some embodiments, the memory 121 mayinclude a non-transitory computable readable medium that is coupled tothe processor 120 and stores one or more executable instructions thatare configured to cause, when executed by the processor 120, theprocessor 120 to perform or implement any of the steps, operations,processes, or methods described herein. In some embodiments, the memory121 is configured to also store, within a database, informationregarding the type, form, or characteristics of the light that eachilluminator 110 is able to or configured to emit and the positions,portions, or objects that each photon detector 111 is configured to orarranged to monitor or detect. The processing circuitry 102 can includeconverters, signal processing circuits, filters, and other interfaceelements in the analog domain and/or the digital domain.

In some embodiments, input/output interface 122 of the processingcircuitry 102 is configured to allow the processing circuitry 102 tocommunicate with the facial tracking device 101 and other devices. Insome embodiments, the input/output interface 122 may be configured toallow for a physical connection (e.g., wired or other physicalelectrical connection) between the processing circuitry 102 and thefacial tracking device 101. In some embodiments, the input/outputinterface 122 may include a wireless interface that is configured toallow wireless communication between the facial tracking device 101(e.g., an ASIC, integrated circuit (IC), or processor on the facialtracking device 101 connected to the illuminator 110 and the photondetector 111) and the processing circuitry 102. The wirelesscommunication may include a Bluetooth, wireless local area network(WLAN) connection, radio frequency identification (RFID) connection, orother types of wireless connections. In some embodiments, theinput/output interface 122 also allows the processing circuitry 102 toconnect to the internet (e.g., either via a wired or wirelessconnection). In some embodiments, the input/output interface 122 alsoallows the processing circuitry 102 to connect to other devices such asa display, audio system, multiple facial tracking devices 101, or otherdevices.

It should be noted that various other components can be included in thefacial tracking system 100 that are not shown for sake of clarity of thepresent embodiments. These can include various power and/or signalconditioning components such as analog to digital converters,multiplexers, transistors, etc. Such additional components can beincluded in either the facial tracking device 101 or the processingcircuitry 102 as appropriate for the particular embodiment.

Referring now to FIG. 2, a view 200 of a user wearing a head mountabledevice integrated with a facial tracking system is shown. The view 200depicts a user 201 having a wearable device 202 (e.g., a head mounteddevise affixed to a head 203 of the user). The wearable device 202includes a facial tracking system 204. The facial tracking system 204may be similar to, or implemented as, the facial tracking system 100described in reference to FIG. 1.

The facial tracking system 204 includes a first facial tracking device240 and a second facial tracking device 245. The first facial trackingdevice 240 is positioned or arranged on or within the wearable device202 such that the first facial tracking device 240 is configured tomonitor or detect movements or positions of objects (e.g., a right sideof a mouth 230 of the user 201) within, at, or along a first portion 290of the head 203. Additionally, the second facial tracking device 245 ispositioned or arranged on or within the wearable device 202 such thatthe second facial tracking device 245 is configured to monitor or detectmovements or positions of objects (e.g., a left side of the mouth 230 ofthe user 201) within, at, or along a second portion 295 of the head 203.In some embodiments, fewer or additional facial tracking devices 240 and245 may be implemented depending upon particular applications.

In some embodiments, the first facial tracking device 240 is mounted toa first side of the wearable device 202 such that the first facialtracking device 240 is arranged to be positioned on a first side of anose 231 of the user and the second facial tracking device 245 ismounted to a second side of the wearable device 202 such that the secondfacial tracking device 245 is arranged to be positioned on a second sideof the nose 231. In this way, the facial tracking system may be able tomonitor a greater amount of area of the head 203 of the user 201 withouthaving obstructions from the nose 231 or other facial features.

The first facial tracking device 240 includes a first illuminator 110 aand a first photon detector 111 a. In some embodiments, the first facialtracking device 240 may include more than one illuminator 110 a and/ormore than one photon detector 111 a. The first illuminator 110 a isconfigured to project or emit a first light at least toward the firstportion 290 and the first photon detector 111 a is configured to receivelight reflected from the first portion 290. The reflected first lightmay then be used to calculate, determine, or estimate a distance ofobjects within the first portion 290 from the first facial trackingdevice 240. For example, the reflected light is received by the photondetector 111 a and the photon detector 111 a generates a correspondingelectrical signal that is indicative of the characteristics of thereflected light. In some embodiments, the electrical signal isindicative of the intensity of the light received (e.g., a highercurrent is generated by the photon detector in response to receiving amore intense light). The facial tracking system 204 (e.g., processingcircuitry 102) may cross-reference within a database or use a formulathat relates distance to the intensity of light received by the photondetector 111 a to an estimated distance of objects within the firstportion 290 from the first facial tracking device 240. In someembodiments, the database or formula may be unique to the type ofilluminator 110 a and photon detector 111 a used and pre-stored withinmemory of the processing circuitry 102.

The second facial tracking device 245 also includes a second illuminator110 b and a second photon detector 111 b. In some embodiments, thesecond facial tracking device 245 may include more than one illuminator110 b and/or more than one photon detector 111 b. The second illuminator110 b is configured to project or emit a second light at least towardthe second portion 295 and the second photon detector 111 b isconfigured to receive light reflected from the second portion 295. Thereflected second light may then be used to calculate, determine, orestimate a distance of objects within the second portion 295 from thesecond facial tracking device 245. For example, the reflected light isreceived by the second photon detector 111 b and the photon detector 111b generates a corresponding electrical signal that is indicative of thecharacteristics of the reflected light. In some embodiments, theelectrical signal is indicative of the intensity of the light received(e.g., a higher current is generated by the photon detector in responseto receiving a more intense light). The facial tracking system 204 maycross-reference within a database or use a formula that relates distanceto the intensity of light received by the second photon detector 111 bto an estimated distance of objects within the second portion 295 fromthe second facial tracking device 245. In some embodiments, the databaseor formula may be unique to the type of illuminator 110 b and photondetector 111 b used and pre-stored within memory of the processingcircuitry 102.

In some embodiments, the facial tracking system 204 also includesprocessing circuitry 102, an integrated circuit (IC), or an ASIC that isat least partially housed within the head mountable device 202. In someembodiments, the processing circuitry 102 also includes processors,memory, or machine readable medium that are located within auxiliarydevices such as server systems, smartphones, personal computers, ordevices that are supplementary to the head mountable device 202. Theprocessing circuitry 102 is communicably coupled to and configured tocontrol the facial tracking devices 240 and 245. For example, theprocessing circuitry 102 is communicably coupled to and configured tocontrol the first illuminator 110 a and the second illuminator 110 b andis communicably coupled to and configured to receive data (raw data,processed data, conditioned data, or otherwise) from the first photondetector 111 a and the second photon detector 111 b.

References to both FIGS. 3 and 4 are made in tandem for purposes ofdemonstration. FIG. 3 depicts an example of a facial tracking device300. FIG. 4 depicts a second example of a facial tracking device 400 isshown. The examples of the facial tracking device 300 and 400 include anilluminator 110 and an array of detectors 301. In some embodiments, theilluminator 110 and the array of detectors 301 are mounted to or withina substrate 302. The substrate 302 may be a portion of a head mountabledevice (e.g., such as wearable device 202 as described in reference toFIG. 2). In some embodiments, the substrate 302 may be a rigid materialsuch as a plastic, metal, composite, fiberglass, or a combinationthereof that is configured to be mechanically affixed selectively orpermanently to a respective device such as a head mountable device. Insome embodiments, the substrate 302 may also have one or more portionsof processing circuitry 102 mounted thereto or within.

The array of detectors 301 include multiple detectors 111 a-h thatenhance the resolution of the facial tracking device 300 or 400 tomonitor or detect one or multiple objects within a respective area(e.g., such as the first portion 290 or the second portion 295 asdescribed in reference to FIG. 2). For example, the illuminator 110 maybe selectively controlled to project a light toward, over, or throughouta first area of a head of a user and each of the multiple detectors 111a-h are arranged and configured to receive portions of the lightreflected from objects within respective portions of the first area. Thelight received by each of the multiple detectors 111 a-h may then beprocessed or used to determine a landscape or facial expression of thefirst area.

The exact configuration of the array of detectors 301 (e.g., or numberof multiple detectors 111 a-h within the array of detectors 301) maydepend upon the particular application or wearable device that therespective facial tracking device 300 or 400 is to be integrated with.For example, now referring to FIG. 3, in some embodiments, the multipledetectors 111 a-h may be arranged linearly. In such an example, themultiple detectors 111 a-h may allow for the facial tracking device 300to monitor or detect positions of objects along an axis of a first area.In another example, now referring to FIG. 4, the multiple detectors 111a-h may be arranged along rows and columns. In such an example, themultiple detectors 111 a-h may allow for the facial tracking device 400to monitor or detect positions of objects along a first axis and asecond axis of a first area.

In some embodiments, more than one illuminator 110 may be used with thearray of detectors 301. In some embodiments, the multiple detectors 111a-h may include 2, 3, 4, 5, 6, 7, 8, 9, 10 or more detectors. In someembodiments, the multiple detectors 111 a-h may be arranged in a circle,square, randomly, or any shape that allows for the multiple detectors111 a-h to enhance a resolution of a respective area. In someembodiments, each of the multiple detectors 111 a-h may be configured toreceive light reflected from a respective portion of an area where therespective portions do not overlap. In some embodiments, each of themultiple detectors 111 a-h may be configured to receive light reflectedfrom a respective portion of an area where one or more of the respectiveportions do overlap.

Referring now to FIG. 5, a diagram 500 of a facial tracking devicedetecting an object is shown. The diagram 500 includes a facial trackingdevice 501 detecting or monitoring an object 504. The facial trackingdevice 501 includes an illuminator 110 and a photon detector 111. Insome embodiments, the object 504 may include a facial feature such as aportion of the brow, mouth, nose, or chin of a user.

The illuminator 110 is configured to emit or project a light over afirst area 503 of the object 504. The detector 111 is configured toreceive portions of the light reflected from a second area 502 of theobject. In some embodiments, the first area 503 and the second area 504are the same. In some embodiments, the first area 503 is larger andincludes the second area 504. The detector 111 may then receive theportions of the reflected light and the characteristics of the reflectedlight (e.g., phase, flight time, intensity, frequency, etc.) may then beused to determine a distance that the object within the second area 504is from the facial tracking device 501 (e.g., the detector 111). In someembodiments, the facial tracking device 501 continuously or discretelyin a periodic pre-determined frequency monitors or detects (e.g.,projects and receives reflected light) the object within the second area504. In some embodiments, multiple facial tracking devices are used tomonitor multiple areas of the object 504. In some embodiments, theilluminator is configured to or controlled to emit the light at apre-determined frequency (e.g., in an infrared frequency). Thepre-determined frequency may be selected to reduce possible interferencefrom other sources. For example, a pre-determined frequency in theinfrared range may reduce the likelihood that other light (e.g., notfrom the illuminator) may interfere with the reflected light received bythe photon detectors 111. In some embodiments, time divisionmultiplexing may be used to reduce possible interference from more thanone illuminators. As an example, when there are more than oneilluminators, each illuminator may be turned on at a separate times inshort bursts (e.g., a burst short enough in time such that a particularilluminator is not projecting light when its light is received by thephoton detector) in order to reduce potential interference between thelight from the more than one illuminators. In some embodiments, the timeof flight between illumination and receiving the light at the photondetector may be used to calculate the distance of an object from thefacial tracking device. In some embodiments, frequency divisionmultiplexing may be used in order to reduce possible interferencebetween light from other illuminators. For example, in some embodiments,each illuminator may be configured or controlled to project a light at apre-determined frequency that is different than the other illuminators.

Referring now to FIG. 6, a diagram 600 of a facial tracking system isshown. The diagram 600 includes a facial tracking system 601, a firstexample of a facial expression determination 602, and a second exampleof a facial expression determination 603. The facial tracking system 601includes multiple facial tracking devices 611 a-h. A first set 611 a-dof the multiple facial tracking devices are configured to be positioned(e.g., arranged on a substrate or head mountable device) on a first side(e.g., right side of a nose 640 of a user) of a head of a user and asecond set 611 e-h of the multiple facial tracking devices 611 a-h areconfigured to be positioned on a second side of (e.g., left side of thenose 640) the head of the user.

Each of the multiple facial tracking devices 611 a-h include anilluminator 110 and a photon detector 111 that are arranged andconfigured to monitor or detect a respective portion 612 a-h of anobject (e.g., a mouth 641 of a user). For example, a first facialtracking device 611 a includes an illuminator 110 and a photon detector111. The illuminator 110 of the first facial tracking device 611 a isarranged to project light toward a first area of the object thatincludes a first portion 612 a of the object (e.g., the mouth 641) andthe photon detector 111 is arranged to receive light reflected from thefirst portion 612 a. In this way, the multiple facial tracking devices611 a-h may be used to detect distances of the respective portions 612a-h of the object (e.g., a mouth 641).

The facial tracking device 601 may use the determined or detecteddistances of each portion 612 a-h of the object to determine a facialexpression of the user. That is, the facial tracking system 601 may map,interpolate, graph, or otherwise process the distances of eachrespective portion 612 a-h to make a determination of a facialexpression of the user. For example, the first facial detection device611 a detects that an outer portion (612 a) of the mouth 641 is at afirst distance and a fourth facial detection device 611 d detects that acenter portion (612 d) is at a second distance.

In the first example of a facial expression determination 602, facialtracking system 601 maps out, interpolates, graphs, or otherwiseconstructs an image of the object (e.g., mouth 641) based on themeasured distances of each respective portion 612 a-h and determinesthat the user is frowning. In some embodiments, the facial trackingsystem 601 may determine that the user is frowning based at least on thefirst distance being larger than the second distance by a pre-determinedthreshold. In some embodiments, the pre-determined threshold may be 0.5,0.75, 1, 1.5, 2 or more centimeters.

In the second example of a facial expression determination 603, facialtracking system 601 maps out, interpolates, graphs, or otherwiseconstructs an image of the object (e.g., mouth 641) based on themeasured distances of each respective portion 612 a-h and determinesthat the user is smiling. In some embodiments, the facial trackingsystem 601 may determine that the user is smiling based at least on thefirst distance being smaller than the second distance by apre-determined threshold. In some embodiments, the pre-determinedthreshold may be 0.5, 0.75, 1, 1.5, 2 or more centimeters. In this way,the facial detection system 601 may monitor, detect, or determine afacial expression of a user without a camera or image processing, whichreduces the amount of computational power needed.

Referring now to FIG. 7, a block diagram 700 of circuitry of a facialtracking system is shown. The block diagram 700 includes an array ofdetectors 701, a multiplexer 702, a demodulator 703, an analog todigital converter (ADC) 704, and a processing block 705. The array ofdetectors 701 includes multiple photon detectors each connected to arespective terminal on the multiplexer 702. The multiplexer 702 may havea respective address for each of the multiple detectors such that aprocessor (e.g., at the processing block 705) may be able to determinewhich signals are associated with each of the multiple detectors. Thedemodulator 703 is connected to the multiplexer 702 and configured todemodulate the signals generated by each of the multiple detectors. TheADC 704 is configured to convert the demodulated signals from thedemodulator 703 into digital representations of the demodulated signals.The ADC 704 is connected to the processor (e.g., at the processingblock) that is configured to process the digital representations of thedemodulated signals to determine a facial expression of a user of thefacial tracking system. For example, the processor may use the digitalrepresentations of the demodulated signals to determine a profile ofobjects (e.g., based on the distance of the objects sensed from each ofthe multiple photon detectors) based on the distances sensed by thearray of detectors 701. The processor may then determine based on theprofile of the objects sensed by the array of detectors 702 the facialexpression of the user (e.g., a smile, a frown, a scowl, a smirk, etc.).

References to both FIGS. 8 and 9 are made in tandem for purposes ofdemonstration. FIG. 8 depicts a third example of a facial trackingdevice 800. FIG. 9 depicts a fourth example of a facial tracking device900 is shown. The examples of the facial tracking device 800 and 900include an array of illuminators 801 and a photon detector 111. In someembodiments, the array of illuminators 801 and photon detector 111 aremounted to or within a substrate 802. The substrate 802 may be a portionof a head mountable device (e.g., such as wearable device 202 asdescribed in reference to FIG. 2). In some embodiments, the substrate302 may be a rigid material such as a plastic, metal, composite,fiberglass, or a combination thereof that is configured to bemechanically affixed selectively or permanently to a respective devicesuch as a head mountable device. In some embodiments, the substrate 802may also have one or more portions of processing circuitry 102 mountedthereto or within.

The array of illuminators 801 include multiple illuminators 110 a-h thatare each spatially encoded or configured to emit spatially encodedlight. The detector 111 is configured to receive light reflected from afirst area of an object and each of the multiple illuminators 110 a-hare configured to or arranged to project respective light towardrespective portions of the first area of the object. In this way, thedetector 111 receives light reflected from the first area that includesthe spatially encoded light from at least some of the multipleilluminators 110 a-h. Corresponding circuitry and/or processingcircuitry connected to the detector 111 may then receive data indicativeof light received by the detector 111 and determine a facial expressionof a corresponding user. For example, each of the multiple illuminators110 a-h may be configured to or controlled by a processor to emit lightthat is spatially encoded toward respective portions of an area of anobject. The processor may have stored in a database the respectiveportion that each of the multiple illuminators are arranged to projectlight toward. Thus, the processing circuitry may control each of themultiple illuminators 110 a-h such that each of them emit a light havinga unique characteristic (e.g., frequency, intensity, etc.). Theprocessing circuitry may then determine which portions of the lightreceived by the detector 111 belong to the respective multipleilluminators 110 a-h based at least on the unique characteristics inorder to determine a distance the respective portions of the objectsbased on the spatially encoded light.

The array of illuminators 801 allows the facial tracking devices 800 or900 to be used to map, interpolate, graph, or construct an image of anobject using multiple illuminators and at least one detector. In otherwords, the array of illuminators 801 may allow for enhanced resolutionof the facial tracking devices 800 or 900 in monitoring, detecting, ordetermining a distance profile of a respective area (e.g., such as thefirst portion 290 or the second portion 295 as described in reference toFIG. 2). The exact configuration of the array of illuminators 801 (e.g.,or number of multiple illuminators 110 a-h within the array ofilluminators 801) may depend upon the particular application or headmountable device that the respective facial tracking device 800 or 900is to be integrated with. For example, now referring to FIG. 8, in someembodiments, the multiple illuminators 110 a-h may be arranged linearly.In such an example, the multiple illuminators 110 a-h may allow for thefacial tracking device 800 to monitor or detect positions of objectsalong an axis of a first area. In another example, now referring to FIG.9, the multiple illuminators 110 a-h may be arranged along rows andcolumns. In such an example, the multiple illuminators 110 a-h may allowfor the facial tracking device 900 to monitor or detect positions ofobjects along a first axis and a second axis of a first area.

In some embodiments, more than one detector 111 may be used with thearray of illuminators 801. In some embodiments, the multipleilluminators 110 a-h may include 2, 3, 4, 5, 6, 7, 8, 9, 10 or moreilluminators. In some embodiments, each of the multiple illuminators 110a-h may be configured to project spatially encoded light towardrespective portions of a first area where the respective portions do notoverlap. In some embodiments, each of the multiple illuminators 110 a-hmay be configured to project spatially encoded light toward respectiveportions of a first area where one or more of the respective portions dooverlap. In some embodiments, the respective portions are pre-determinedor pre-measured and stored within a database.

Referring now to FIG. 10, a diagram of a facial tracking system 1000 isshown. The facial tracking system 1000 includes a first facial trackingdevice 1001 and a second facial tracking device 1002 both communicablycoupled to processing circuitry 102. The first facial tracking device1001 is configured to be positioned (e.g., arranged on a substrate orhead mountable device) on a first side (e.g., right side of a nose 1040of a user) of a head of a user and the second facial tracking device1002 is configured to be positioned on a second side of (e.g., left sideof the nose 1040) the head of the user.

The first facial tracking device 1001 includes a first illuminator 110 aand a first set of detectors 111 a-d. The first illuminator 110 a isarranged and controlled by the processing circuitry 102 to project alight onto a first area of a head of a user. For example, the first areamay include a right side of a mouth of the user. Each of the first setof detectors 111 a-d are arranged to receive light reflected fromrespective portions of the first area. For example, a first detector 111a may be arranged to receive light reflected from a first portion of thefirst area that corresponds to an outer edge of a mouth of the user(e.g., an outer right edge of lips or mouth), a second detector 111 bmay be arranged to receive light reflected from a second portion of thefirst area that corresponds to a position of the mouth more center thanthe first portion, and a fourth detector 111 d may be arranged toreceive light reflected from a position that is near the center of themouth (e.g., and/or near the center of the mouth on the same side of themouth as the first portion). The second facial tracking device 1002includes a second illuminator 110 b and a second set of detectors 111e-h. The second illuminator 1002 and the second set of detectors 111 e-hmay be arranged and configured to perform similar but symmetricoperations on a second area of the head. For example, the second areamay include the left side of the mouth.

Each of the detectors 111 a-h are configured to receive light fromreflected portions of a head in some embodiments. The exact position andor arrangements of each detector 111 a-h and corresponding portions ofthe head may be known or determined and stored within a database.

Referring now to FIG. 11, a schematic diagram 1100 of a winner take allcircuit 1101 and example outputs are shown. In some embodiments, afacial tracking system such as the facial tracking system 1000 describedin reference to FIG. 10 may determine a facial expression of a user inthe analog domain. For example, a facial tracking system may use awinner take all circuit or similar circuit to determine a facialexpression of a user based on an output of the circuit.

The schematic diagram 1100 includes a winner take all circuit 1101. Thewinner take all circuit 1101 includes a first branch 1110 and a secondbranch 1111 each connected to a bias current source 1112. In someembodiments, the winner take all circuit 1101 has as a branch for eachone of the detectors 111 of a corresponding facial tracking device. Eachbranch includes a corresponding detector 110 a, a current source 1115, afirst transistor 1116, and a second transistor 1117. An output of thecurrent source 1115 is connected to an output of the detector 110 a anda first terminal of the first transistor 1116. A second terminal of thefirst transistor 1116 is connected to ground. A control terminal (e.g.,gate terminal) of the first transistor 1116 is connected to a firstterminal of the second transistor 1117 and to an input terminal of thebias current source 1112. A control terminal (e.g., a gate terminal ofthe second transistor 1117 is connected to the first terminal of thefirst transistor 1116. A second terminal of the second transistor 1117is connected to a processor of the processing circuitry 102. In someembodiments, the second terminal of the second transistor 1117 isconnected to a respective input of a multiplexer or other addressspecific device.

The winner take all circuit 1101 is configured to signal to theprocessor a detector that has the highest or most intense signal. Theprocessing circuitry may then determine a facial expression of the userbased on which of the branches (e.g., and corresponding detector) hasthe highest or most intense signal. For example, the highest or mostintense signal may indicate that the respective detector has an objectcloser to the respective detector than any other of the diodes. Thus,referring now back to FIG. 10, if a branch that corresponds to the firstdiode 111 a is outputting the signal, the processing circuitry maydetermine that the portion corresponding to the first diode 111 a (e.g.,the outer edge of the mouth) has the smallest distance. The processormay then determine that the facial expression of the user is a smile(e.g., because the outer edge of the mouth is higher or closer to thedetectors than the center of the mouth.).

In some embodiments, each facial tracking device may be connected to acorresponding winner take all circuit 1101. In some embodiments, two ormore of the facial tracking devices may be connected to a correspondingwinner take all circuit 1101. The outputs of the winner take allcircuits 1101 may be connected to a processor directly or via circuitrysuch as a multiplexer. The multiplexer may be configured to signal tothe processing circuitry an address of the branch or branches that areoutputting a signal. The processing circuitry may cross reference theaddress or branch location within a database to determine which positionon the head that the address corresponds to and determine the facialexpression based on the corresponding position of the head. For example,if a branch corresponding to the detectors 111 d on one winner take allcircuit is outputting a signal and a branch corresponding to thedetector 111 e on another winner take all circuit is outputting asignal, then the processor may determine that the facial expression ofthe user is a frown.

Referring now to FIG. 12, a flow diagram 1200 of a method of facialtracking is shown. In operation 1201, the facial tracking systemprojects light, using an illuminator, toward a face of a user. In someembodiments, a processor connected to the one or more illuminatorscauses the one or more illuminators to each project a light. In someembodiments, the facial tracking system causes the one or moreilluminators to each project a light toward respective portions of ahead (e.g., face or mouth) of a user). In some embodiments, the facialtracking system causes each of the illuminators to project spatiallyencoded light toward respective portions of the head. In someembodiments, the facial tracking system causes each of the one or moreilluminators to project light having the same characteristics towardrespective portions of the head. In some embodiments, the respectiveportions may overlap with one another. In some embodiments, therespective portions may not overlap with another.

In operation 1202, the facial tracking system receives data indicativeof received light from one or more photon detectors. That is, the lightreflected from the respective portions of the head is received by one ormore photon detectors and the detectors generate electrical signalsindicative of the received light. In some embodiments, the electricalsignals are received by a winner take all circuit and the winner takeall circuit outputs a signal to the processor. In some embodiments, theelectrical signals are converted into digital form (e.g., via an ADC)and received by the processing circuitry. In some embodiments,additional processing of the electrical signals via analog and/ordigital circuitry may occur before the electrical signals are receivedby the processing circuitry.

In operation 1203, the facial tracking system reconstructs spatialinformation based on the data of the one or more photon detectors. Thatis, the processing circuitry reconstructs a profile of the face based onthe electrical signals. The electrical signals are reconstructed tocreate the profile of the face. In some embodiments, such as when theprocessing circuitry receives a signal from a winner take all circuit,the processor determines which portion of the face is closest or nearestto a particular detector (e.g., based on an address of the signalassociated with the particular detector). The processing circuitry thenmay cross reference within a database the particular detector (e.g., theaddress) with a location and a respective portion of the face that theparticular detector is positioned to receive light from. In this way,the processing circuitry may reconstruct the profile of the face basedon the signal.

In some embodiments, the facial tracking system may determine a distanceof objects sensed by each of the detectors using the electrical signals.The distance of objects sensed by each of the detectors may bedetermined based on the intensity (e.g., intensity or magnitude ofcurrent) generated by the photon detector. The intensity or magnitude ofthe signal (e.g., current) generated by each of the photon detectors maythen be cross-referenced or used within a formula of known intensitiesor magnitudes of the signal versus a distance of the object from thephoton detector in order to determine the distances. The processingcircuitry may then use the distances, known address of each of thedetectors, and the known respective portions of the head that each ofthe detectors are configured to receive reflected light from to create aprofile of the face. In some embodiments, the facial tracking system mayreceive a signal from a photon detector that includes received lightfrom a portion of the head and reconstruct the profile of the face basedon the characteristics of the light received by the photon detector(e.g., and encoded within the signal). For example, in an embodimentwhere multiple illuminators are used, each being configured to projectspatially encoded light toward respective portions of the face, thephoton detector may output a signal encoded with information of eachspatially encoded light received by the photon detector and theprocessing circuitry may reconstruct a profile based on the signal andknown characteristics and positioning of the spatially encoded light.

In operation 1204, the facial tracking system determines a facialexpression based on the reconstructed spatial information. In someembodiments, the processing circuitry uses the reconstructed profile todetermine a facial expression on the face of the user. For example, insome embodiments, the profile may be a profile of a mouth of the userand the processing circuitry may determine that the profile indicatesthat the user is smiling, scowling, frowning, or smirking. In someembodiments, the profile bay be a profile of a brow or forehead of theuser and the profile may indicate that the user is expressing a worriedlook, or an angry look based on the profile of the brow.

Having now described some illustrative implementations, it is apparentthat the foregoing is illustrative and not limiting, having beenpresented by way of example. In particular, although many of theexamples presented herein involve specific combinations of method actsor system elements, those acts and those elements can be combined inother ways to accomplish the same objectives. Acts, elements andfeatures discussed in connection with one implementation are notintended to be excluded from a similar role in other implementations orimplementations.

The hardware and data processing components used to implement thevarious processes, operations, illustrative logics, logical blocks,modules and circuits described in connection with the embodimentsdisclosed herein may be implemented or performed with a general purposesingle- or multi-chip processor, a digital signal processor (DSP), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A generalpurpose processor may be a microprocessor, or, any conventionalprocessor, controller, microcontroller, or state machine. A processoralso may be implemented as a combination of computing devices, such as acombination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. In some embodiments, particularprocesses and methods may be performed by circuitry that is specific toa given function. The memory (e.g., memory, memory unit, storage device,etc.) may include one or more devices (e.g., RAM, ROM, Flash memory,hard disk storage, etc.) for storing data and/or computer code forcompleting or facilitating the various processes, layers and modulesdescribed in the present disclosure. The memory may be or includevolatile memory or non-volatile memory, and may include databasecomponents, object code components, script components, or any other typeof information structure for supporting the various activities andinformation structures described in the present disclosure. According toan exemplary embodiment, the memory is communicably connected to theprocessor via a processing circuit and includes computer code forexecuting (e.g., by the processing circuit and/or the processor) the oneor more processes described herein.

The present disclosure contemplates methods, systems and programproducts on any machine-readable media for accomplishing variousoperations. The embodiments of the present disclosure may be implementedusing existing computer processors, or by a special purpose computerprocessor for an appropriate system, incorporated for this or anotherpurpose, or by a hardwired system. Embodiments within the scope of thepresent disclosure include program products comprising machine-readablemedia for carrying or having machine-executable instructions or datastructures stored thereon. Such machine-readable media can be anyavailable media that can be accessed by a general purpose or specialpurpose computer or other machine with a processor. By way of example,such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to carry or storedesired program code in the form of machine-executable instructions ordata structures and which can be accessed by a general purpose orspecial purpose computer or other machine with a processor. Combinationsof the above are also included within the scope of machine-readablemedia. Machine-executable instructions include, for example,instructions and data which cause a general purpose computer, specialpurpose computer, or special purpose processing machines to perform acertain function or group of functions.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including” “comprising” “having” “containing” “involving”“characterized by” “characterized in that” and variations thereofherein, is meant to encompass the items listed thereafter, equivalentsthereof, and additional items, as well as alternate implementationsconsisting of the items listed thereafter exclusively. In oneimplementation, the systems and methods described herein consist of one,each combination of more than one, or all of the described elements,acts, or components.

Any references to implementations or elements or acts of the systems andmethods herein referred to in the singular can also embraceimplementations including a plurality of these elements, and anyreferences in plural to any implementation or element or act herein canalso embrace implementations including only a single element. Referencesin the singular or plural form are not intended to limit the presentlydisclosed systems or methods, their components, acts, or elements tosingle or plural configurations. References to any act or element beingbased on any information, act or element can include implementationswhere the act or element is based at least in part on any information,act, or element.

Any implementation disclosed herein can be combined with any otherimplementation or embodiment, and references to “an implementation,”“some implementations,” “one implementation” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described in connectionwith the implementation can be included in at least one implementationor embodiment. Such terms as used herein are not necessarily allreferring to the same implementation. Any implementation can be combinedwith any other implementation, inclusively or exclusively, in any mannerconsistent with the aspects and implementations disclosed herein.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded to increase the intelligibility of the drawings, detaileddescription, and claims. Accordingly, neither the reference signs northeir absence have any limiting effect on the scope of any claimelements.

Systems and methods described herein may be embodied in other specificforms without departing from the characteristics thereof. Furtherrelative parallel, perpendicular, vertical or other positioning ororientation descriptions include variations within +/−10% or +/−10degrees of pure vertical, parallel or perpendicular positioning.References to “approximately,” “about” “substantially” or other terms ofdegree include variations of +/−10% from the given measurement, unit, orrange unless explicitly indicated otherwise. Coupled elements can beelectrically, mechanically, or physically coupled with one anotherdirectly or with intervening elements. Scope of the systems and methodsdescribed herein is thus indicated by the appended claims, rather thanthe foregoing description, and changes that come within the meaning andrange of equivalency of the claims are embraced therein.

The term “coupled” and variations thereof includes the joining of twomembers directly or indirectly to one another. Such joining may bestationary (e.g., permanent or fixed) or moveable (e.g., removable orreleasable). Such joining may be achieved with the two members coupleddirectly with or to each other, with the two members coupled with eachother using a separate intervening member and any additionalintermediate members coupled with one another, or with the two memberscoupled with each other using an intervening member that is integrallyformed as a single unitary body with one of the two members. If“coupled” or variations thereof are modified by an additional term(e.g., directly coupled), the generic definition of “coupled” providedabove is modified by the plain language meaning of the additional term(e.g., “directly coupled” means the joining of two members without anyseparate intervening member), resulting in a narrower definition thanthe generic definition of “coupled” provided above. Such coupling may bemechanical, electrical, or fluidic.

References to “or” can be construed as inclusive so that any termsdescribed using “or” can indicate any of a single, more than one, andall of the described terms. A reference to “at least one of ‘A’ and ‘B’”can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Suchreferences used in conjunction with “comprising” or other openterminology can include additional items.

Modifications of described elements and acts such as variations insizes, dimensions, structures, shapes and proportions of the variouselements, values of parameters, mounting arrangements, use of materials,colors, orientations can occur without materially departing from theteachings and advantages of the subject matter disclosed herein. Forexample, elements shown as integrally formed can be constructed ofmultiple parts or elements, the position of elements can be reversed orotherwise varied, and the nature or number of discrete elements orpositions can be altered or varied. Other substitutions, modifications,changes and omissions can also be made in the design, operatingconditions and arrangement of the disclosed elements and operationswithout departing from the scope of the present disclosure.

References herein to the positions of elements (e.g., “top,” “bottom,”“above,” “below”) are merely used to describe the orientation of variouselements in the FIGURES. The orientation of various elements may differaccording to other exemplary embodiments, and that such variations areintended to be encompassed by the present disclosure.

What is claimed is:
 1. A head wearable display (HWD) comprising: anelement configured to be worn on a head of a user; an illuminatordisposed with the element and configured to emit light toward at least aportion of the head; a photon detector disposed with the element andconfigured to receive light reflected from at least a second portion ofthe head; one or more processors coupled to a non-transitorycomputer-readable storage medium having instructions encoded thereonthat, when executed by the one or more processors, cause the one or moreprocessors to: cause the illuminator to project the light toward theportion of the head; receive information from the photon detector; anddetermine a facial expression of the user based at least on theinformation from the photon detector.
 2. The HWD of claim 1, wherein theportion of the head and the second portion of the head overlap at leastpartially.
 3. The HWD of claim 1, further comprising more than onephoton detector, wherein the more than one photon detectors are arrangedin an array and configured to receive light reflected from respectiveportions of the head.
 4. The HWD of claim 3, wherein to determine thefacial expression, the non-transitory computer-readable storage mediumhaving instructions encoded thereon that, when executed by the one ormore processors, cause the one or more processors to further receiveinformation from each of the more than one photon detectors.
 5. The HWDof claim 1, further comprising more than one illuminator, wherein themore than one photon detectors are arranged in an array and eachconfigured to emit light with spatially encoded characteristics towardrespective portions of the head.
 6. The HWD of claim 5, wherein theinformation from the photon detector comprises characteristics of thelight received by the photon detector, and wherein the element is aglass frame or a housing.
 7. The HWD of claim 6, wherein the first andsecond portions of the head are on one side of a nose of the user.
 8. Afacial tracking device comprising: an illuminator configured to emitlight toward at least a portion of a face of a user; a photon detectorconfigured to receive light reflected from at least a second portion ofthe face; and a processor configured to: receive information regardingthe characteristics of light detected by the photon detector; anddetermine a facial expression of the face based on the characteristicsof light detected by the photon detector.
 9. The facial tracking deviceof claim 8, further comprising circuitry configured to indicate a facialexpression of the face based on characteristics of light received by thephoton detector.
 10. The facial tracking device of claim 9, wherein thecircuitry comprises a winner take all circuit configured to output asignal to the processor that is indicative of the facial expression ofthe face to the processor.
 11. The facial tracking device of claim 8,further comprising more than one photon detector, wherein the more thanone photon detectors are arranged in an array and configured to receivelight reflected from respective portions of the head.
 12. The facialtracking device of claim 11, wherein the respective portions of the headare all located on a first side of a nose of the face, and wherein thefacial tracking device further comprises: a second illuminator emitlight toward at least a portion of a second side of the face; a secondset of more than one photon detectors, wherein the second set isarranged in a second array and configured to receive light reflectedfrom respective portions of the second side of the face.
 13. The facialtracking device of claim 12, wherein to determine the facial expressionthe processor is further configured to receive information regardingcharacteristics of the light from the more than one photon detectors andthe second set of the more than one photon detectors.
 14. The facialtracking device of claim 8, wherein the portion of the face and thesecond portion of the face overlap at least partially.
 15. The facialtracking device of claim 8, wherein characteristics of the light of theilluminator are spatially encoded corresponding to the portion of theface of the user.
 16. A method of tracking a facial expression of a usercomprising: projecting light using an illuminator of a facial trackingsystem toward a face of a user; receiving, via the facial trackingsystem, information from a photon detector, the information comprisingcharacteristics of the light reflected from the face; and determining,via the facial tracking system, a facial expression of the face based onthe information from the photon detector.
 17. The method of claim 16,wherein turning on the illuminator further comprises turning on multipleilluminators of the facial tracking system such that each of themultiple illuminators project a light having different characteristicstoward respective portions of the face.
 18. The method of claim 17,wherein the determining the facial expression further comprisesreconstructing an image of at least a portion of the face based on theinformation from the photon detector.
 19. The method of claim 16,wherein receiving the information from the photon detector furthercomprises receiving, via the facial tracking system, information frommultiple photon detectors, wherein each of the multiple photon detectorsare configured to receive light from respective portions of the face.20. The method of claim 19, wherein the determining the facialexpression further comprises reconstructing an image of at least aportion of the face based on the information from the multiple photondetectors.